Harvesting labels for significant locations and updating a location fingerprint database using harvested labels

ABSTRACT

This disclosure describes embodiments for harvesting and serving labels for locations. In an embodiment, a method comprises: receiving, by one or more server computers, location data including wireless access point data and location labels associated with significant locations, the location data being harvested from a plurality of devices operating at a plurality of geographic locations; and updating, by the one or more server computers, a plurality of fingerprints representing the plurality of geographic locations, the updating including associating at least one of the received location labels with at least one of the plurality of fingerprints.

TECHNICAL FIELD

This disclosure relates generally to location-based services.

BACKGROUND

A map service can provide maps to one or more computing devices. Themaps can include points of interest (POIs). A POI can be a place that isdesignated as useful or of interest to any user or all users. Forexample, a POI can be a shop, restaurant, or a hotel. Each POI may havean address (for example, a street address). To display a POI in a map,the map service can geocode the POI by associating a location (forexample, latitude and longitude coordinates) with the POI. The onlinemap service can display a marker representing the POI on the map at thelatitude and longitude coordinates. The inverse technique, reversegeocoding, attempts to determine a POI to which a given latitude andlongitude refers.

SUMMARY

This disclosure describes embodiments for harvesting and serving labelsfor locations.

In an embodiment, a method comprises: receiving, by one or more servercomputers, location data including wireless access point data andlocation labels associated with significant locations, the location databeing harvested from a plurality of devices operating at a plurality ofgeographic locations; and updating, by the one or more server computers,a plurality of fingerprints representing the plurality of geographiclocations, the updating including associating at least one of thereceived location labels with at least one of the plurality offingerprints.

In an embodiment, a method comprises: receiving, by one or more servercomputers, wireless access point data obtained by a requesting deviceoperating at a significant location; comparing, by the one or moreserver computers, the wireless access point data with a plurality offingerprints; responsive to the comparing, obtaining, by the one or moreserver computers, a matching fingerprint; obtaining, by the one or moreserver computers and using the matching fingerprint, a location labelfor the significant location; and sending, by the one or more servercomputers to the requesting device, the location label for thesignificant location.

In an embodiment, a method comprises: obtaining, by a computing device,wireless access point data; obtaining, by the computing device, asignificant location; obtaining, by the computing device, a label forthe significant location; storing, by the computing device, the wirelessaccess point data and location label; responsive to a trigger eventdetected by the computing device: collecting the stored wireless accesspoint data and location label; and sending the collected wireless accesspoint data and location label to one or more server computers.

In an embodiment, a system comprises: one or more processors; memorycoupled to the one or more processors and operable for storinginstructions, which, when executed by the one or more processors, causesthe one or more processors to perform operations comprising: receivinglocation data including wireless access point data and location labelsassociated with significant locations, the location data being harvestedfrom a plurality of devices operating at a plurality of geographiclocations; and updating a plurality of fingerprints representing theplurality of geographic locations, the updating including associating atleast one of the received location labels with at least one of theplurality of fingerprints.

In an embodiment, a system comprises: one or more processors; memorycoupled to the one or more processors and operable for storinginstructions, which, when executed by the one or more processors, causesthe one or more processors to perform operations comprising: receivingwireless access point data obtained by a requesting device operating ata significant location; comparing the wireless access point data with aplurality of fingerprints; responsive to the comparing, obtaining amatching fingerprint; obtaining, using the matching fingerprint, alocation label for the significant location; and sending to therequesting device the location label for the significant location.

In an embodiment, a system comprises: one or more processors; memorycoupled to the one or more processors and operable for storinginstructions, which, when executed by the one or more processors, causesthe one or more processors to perform operations comprising: obtainingwireless access point data; obtaining a significant location; obtaininga label for the significant location; storing the wireless access pointdata and location label; responsive to a trigger event detected by thecomputing device: collecting the stored wireless access point data andlocation label; and sending the collected wireless access point data andlocation label to one or more server computers.

Particular embodiments disclosed herein provide one or more advantages.For example, the harvesting of location labels allows for the creationof a network-based service for client computing devices that do not havelabeled data for a significant location. The harvested data is uploadedanonymously meaning that the harvested data does not include informationthat identifies the user of the client computing device being harvested.Additionally, the user may opt out of the harvesting using, for example,a setting pane or menu implemented on the computing device.

The details of one or more embodiments of the subject matter describedin this specification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages of thesubject matter will become apparent from the description, the drawings,and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of an example virtual map presented by acomputing device showing POIs and a significant location.

FIG. 2A is a schematic diagram illustrating multiple points of interestnear a significant location.

FIG. 2B is a block diagram illustrating components of an example systemfor implementing labeling of significant locations based on contextualdata.

FIG. 3A is a block diagram illustrating contextual data sources used toidentify labels for significant locations.

FIG. 3B is a schematic diagram illustrating two geographic areasassociated with a significant location encompassing POIs.

FIG. 4 is a flowchart of an example process for identifying labels forsignificant locations.

FIG. 5 is a conceptual diagram of an example system for harvestinglabels for locations.

FIG. 6 is a flowchart of an example process for harvesting labels forlocations performed by a computing device.

FIG. 7 is a flowchart of an example process for harvesting labels forlocations performed by a server computer(s).

FIG. 8 is a flowchart of an example process for serving labels forlocations performed by a server computer(s).

FIG. 9 is a conceptual block diagram of an example system for harvestingand serving labels for locations.

FIG. 10 is a block diagram of an example device architecture forimplementing the features and processes described in reference to FIGS.1-9.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION Exemplary Use Cases

FIG. 1 is a schematic diagram of an example map presented on a displayof a computing device. In the example shown, the computing device (e.g.,a smartphone, tablet computer, wearable device, desktop computer orother computing device) executes a map application that displays a mapof a geographic area including three streets: Main Street (goingnorth-south), and cross streets First Street and Second Street, eachgoing east-west. The map application overlays on the map POI markers123, 124 representing a “MOVIE THEATER” and a “BURGER PLACE,”respectively. In general, a POI can be a business, landmark, publicpark, school or any other entity that is of interest to or is meaningfulto a user or all users.

In addition to POI markers 123, 124, the map application also displays amarker 122 representing a location of the computing device. In someinstances, the location of the computing device can be any geographiclocation at which the computing device is physically located. In anexample in which the computing device is a smartphone or a wearabledevice (for example, a smartwatch) or an automobile information andentertainment center, the marker 122 can be displayed at a location of auser possessing the smartphone. In some instances, the location of thecomputing device can be a location that is significant to the user ofthe computing device. For example, a significant location can be theuser's home or work address or a location that the user has visitedseveral times in the past (e.g., a frequently visited business).

Determining that a location is a significant location can includedetermining that the computing device dwells at the significant locationfor at least a threshold amount of time. As described below, thesignificant location can be inferred from clusters of latitude andlongitude readings from, for example, satellite-based orterrestrial-based positioning technologies, such as a Global NavigationSatellite System (GNSS) receiver, which were gathered previously. Assuch, the significant location may have associated geographicuncertainty—that is, its boundaries may not be precisely known. Theassociations between its cluster of latitude and longitude readings andone or more POIs may also not be known.

Therefore, a significant location may be represented by a marker on themap with a label that is not meaningful to the user. For example, asignificant location for the user may be a frequently visitedrestaurant, such as BURGER PLACE 124. The user would expect to see thesignificant location marker 122 located at, for example, the entrance toBURGER PLACE 124 with an appropriate label describing the restaurant.However, since significant locations are estimated (e.g., using aclassifier or machine learning), the location marker 122 is in themiddle of Main Street with a generic label (e.g., only a street address)that has no meaning to the user. This results in a poor user experiencewith the map application. By labeling the geographic location 122 with alabel that is meaningful to the user (e.g., “BURGER PLACE”), the mapapplication personalizes the significant location to the user, therebyenhancing the user's experience with the map application.

Implementations of the subject matter described in this disclosure canenhance operation of a computing device by using contextual data storedon or accessible by the computing device to improve the usefulness ofsignificant locations to a user. Contextual data can include data uniqueto the computing device or personalized for a user of the computingdevice (or both). The contextual data can include data collected basedon the user's usage of the computing device, for example, based onexecuting, using the computing device, one or more applications that thecomputing device is configured to execute.

In some implementations, the contextual data collected based on usage ofthe computing device can include any input provided to the computingdevice to execute any application. The input can be expressly providedby the user or can be inherently obtained in response to the userperforming an action. The contextual data can include any outputprovided by the computing device either in response to any input or inresponse to executing any application.

The contextual data can include any information derived or inferred bythe computing device based on usage (present or past usage or both) ofthe computing device, in an environment in which the computing device isor was present, based on information received from the computing devicefrom a source other than the user (for example, a central server, a WiFiservice provider, a telephony service provider or other source) or anycombination of them. For example, by leveraging contextual data that isalready being collected for other applications running on the computingdevice to label significant locations, the utility of significantlocations to the user is improved without expending additional resourcesto collect or derive additional data. This results in an improvement tothe computing device by preserving memory and storage and reducing powerconsumption by the computing device by not performing additional datacollection and processing.

Exemplary Labeling of Significant Locations Based on Contextual Data

FIG. 2A is a schematic diagram illustrating POIs near a significantlocation 270. The schematic diagram illustrates that multiple POIs canfall within an uncertainty associated with a significant location. Inthe example shown, the significant location 270 is located among 8 POIs(POI 1-POI 8). The uncertainty associated with the significant location270 defines a geographic area of uncertainty 272 that encompasses asubset of the POIs (for example, POIs 2, 3, 4, 5, 6 and 7). In someimplementations, the computing device 200 can store or derive contextualdata associated with at least one of the subset of the POIs encompassedby the geographic area of uncertainty 272. For example, the user of thecomputing device 200 may have executed a payment transaction using thecomputing device 200 at POI 3. Contextual details associated with thepayment transaction (e.g., timestamp, merchant name and address or otherbusiness identifier) can be stored on the computing device 200 duringthe transaction. The contextual details associated with paymenttransactions can further show that the user has not executed a paymenttransaction at any of the other POIs in the subset of POIs over aspecified time period.

In another example, a disambiguation based on crowd-sourced paymenthistory can be used to identify POI 3. For example, several users ofrespective computing devices may have executed respective paymenttransaction using their respective computing devices at POI 3.Contextual details associated with each payment transaction (e.g.,timestamp, merchant name and address or other business identifier) canbe stored on respective computing devices during respectivetransactions. The contextual details associated with paymenttransactions can further show that some or all the users have notexecuted payment transactions at any of the other POIs in the subset ofPOIs over a specified time period.

In a further example, the user of the computing device 200 may havefrequented POI 3 multiple times over a duration (for example, a week, amonth). The computing device 200 can store a dwell time of the computingdevice 200 at POI 3 or a map route to the POI 3. The number of visitscan be tracked with a software counter. The dwell time can be determinedfrom a motion sensor (e.g., accelerometer) or GNSS receiver on thecomputing device 200 and a timer. Based on such contextual details (orother contextual details such as those described below), the computingdevice 200 can determine that, out of all the POIs in the subset, POI 3is most likely to be the significant location. For example, based onmultiple visits to the same geographic location and multiple paymenttransactions at POI 3 during each of those visits, the computing device200 can determine that POI 3 is most likely to be the significantlocation. The maps engine 206 can identify a label (e.g., a merchantname) associated with POI 3. For example, the label can be stored in thePOI database 210. The identified label is meaningful to the user of thecomputing device 200 because the user has visited POI 3 one or moretimes in the past.

FIG. 2B is a block diagram illustrating components of an exemplarycomputing device 200 implementing labeling of locations, for example,significant locations or any location, based on contextual dataassociated with or stored on (or both) the computing device. Returningto the example described above with reference to FIG. 2A, byimplementing the techniques described below with reference to FIG. 2B,the location of the computing device 200 can be resolved to POI 3 fromamong the subset of POIs, namely, POIs 2, 3, 4, 5, 6 and 7. Thecomputing device 200 can include one or more processors and acomputer-readable storage medium storing instructions executable by theone or more processors to perform operations described herein.Additional details about the computing device 200 architecture aredescribed below with reference to FIG. 5. The computing device 200includes multiple components (for example, multiple modules), each ofwhich can be implemented as computer instructions executable by the oneor more processors, as described in reference to FIG. 5.

In some implementations, the computing device 200 includes a significantlocation estimator 202. The significant location estimator 202 canoutput significant locations 204. Each significant location can beassociated with an entity POI that is located at a geographic location.Each POI is assumed to have meaning to any user of any computing deviceor all users of respective computing devices. For example, a significantlocation can include the user's home or work place or other locationthat the user of the computing device frequently visits. For example, asignificant location can include a business such as the user's favoriterestaurants, movie theaters, or other shops.

In some implementations, the computing device 200 includes a locationsource 203 that can output locations 205. Each location output by thelocation source 203 can be associated with a POI that is located at ageographic location. A location output by the location source 203 may ormay not have significance to the user of the computing device.Therefore, a location output by the location source 203 may overlap witha location output by the significant location estimator 202.

In some implementations, each location (e.g., location output by thesignificant location estimator 202 or the location source 203) can berepresented by an estimated geographic location and data representing ameasure of uncertainty or error in the estimated location. Thegeographic location can be identified by position coordinates in anyreference coordinate frame such as a geodetic or East North Up (ENU)reference coordinate frame. In an implementation, the geographiclocation is represented by latitude, longitude and altitude. Theuncertainty or error can be a statistical variance associated with theestimated significant location. In an implementation, the error can berepresented on a map as a geographic area (e.g., a circle or otherpolygon) surrounding the estimated significant location. The exactlocation of the significant location can be anywhere within thegeographic area. In some implementations, the significant locationestimator 202 and the location source 203 can generate, output and storea list of significant locations 204 and a list of locations 205,respectively. Over time, the significant location estimator 202 canupdate the list, for example, by adding more significant locations orremoving locations that are no longer significant (e.g., locations thathave not been visited for a long period of time). In an implementation,the list is an ordered list with the significant location at the top ofthe list having the lowest uncertainty or error. Similarly, over time,the locations in the location source 203 can also be updated.

The computing device 200 can include a maps engine 206 connected to amaps database 208 and a POI database 210. The maps engine 206 is alsoaccessible by computing device applications 218 a-218 d through, forexample an application programming interface (API). The computing deviceapplications 218 a-218 d can request map data from maps engine 206 thatcan be rendered into a map. In an implementation, the maps database 208and POI database 210 can be a single database. The maps database 208maintains map data for generating a map including data for renderingstreets, highways, freeways and the like on the map. The POI database210 maintains a list of entities (e.g., businesses, public landmarks,parks, schools, hospitals) that can be represented by markers (e.g.,virtual pushpins) overlaid on the map. For example, the maps database208 can store a street address of an entity and the POI database 210 canstore information identifying the entity as a particular restaurant,which can serve as the label for the entity. In particular, POI database210 can store one or more labels for each of the entities. The one ormore labels can include, for example, a business name of an entityprovided by a POI service, a tag entered by a user for the entity, or acontact name associated with the entity in a contact list (e.g., a homeor work address). The maps database 208 and the POI database 210 canprovide the list of entities and their identities to the maps engine206. The maps engine 206 can display markers representing thesignificant locations in the map together with the markers for the POIs.

The computing device 200 includes a context analyzer 212 configured tocollect and analyze contextual data to identify a label for a location(e.g., a location output by the significant location estimator 202 or alocation output by the location source 203) at or near which thecomputing device is located. In some implementations, a location processin the computing device 200 can obtain (e.g., receive or estimate) ageographic location of the computing device 200. For example, thelocation processor can receive or derive the geographic location of thecomputing device from signals transmitted by a terrestrial-based orsatellite-based location estimation system, such as, for example, a WiFistation, cell tower or GNSS. An example GNSS system is the GlobalPositioning System (GPS). The geographic location of the computingdevice can be represented by a latitude, longitude and altitude. Theestimated geographic location of the computing device can also includean uncertainty, which can be represented by a geographic area (e.g.,circular region or other polygon) surrounding the estimated geographiclocation of the computing device. In some implementations, the contextanalyzer 212 can generate and output a list of significant locations 214that includes, for each significant location, location information (forexample, latitude or longitude or altitude or any combination of them,and an uncertainty) and a maps unique identifier (MUID) (describedbelow).

The maps engine 206 can obtain POIs associated with, for example, near asignificant location. As described earlier, each POI represents anentity within or near a geographic area of uncertainty surrounding theestimated significant location. For example, the maps engine 206 canprovide the POIs, which are stored in the POI database 210 or retrievedfrom a network-based POI service if available. The geographic locationsof the POIs can reside inside or outside the geographic area ofuncertainty associated with the estimated significant location. The mapsengine 206 can determine that at least one of the POIs is a significantlocation by comparing the geographic locations of the POIs with thegeographic locations of the significant locations in the list ofsignificant locations 204 provided by the significant location estimator202. If a significant location is the same as a POI location, then thesignificant location can be replaced by the POI or can assume the labelof the matching POI. If, however, there is no match, the contextanalyzer 212 can determine that one of the POIs is a significantlocation based on contextual data associated with the POIs, as describedwith reference to FIG. 2B.

FIG. 3A is a block diagram illustrating contextual data sources 214 usedto identify labels for significant locations. The contextual datasources 214 can include multiple components, each storing contextualdata which the context analyzer 212 can use to identify a labelassociated with a significant location. Several examples of contextualdata for identifying labels associated with significant locations aredescribed below. If the label is associated with a POI in the POIdatabase 210, the POI can be associated with a POI identifier or a mapunique identifier (MUID). When a computing device application 218 a-218d running on the computing device 200 requests a significant locationthat can be mapped to a nearby POI, the MUID for the POI can be providedto the requesting application. The requesting application can then sendthe MUID to the maps engine 206 (e.g., through an API), which can beused by the maps engine 206 to retrieve the label from the POI database210.

In the examples for identifying a label described below, an identifiedlabel is described as being displayed in a user interface. In someimplementations, a confidence level for the identified label can bedetermined and displayed in the user interface together with the label.The confidence level can represent a likelihood that the identifiedlabel matches the accurate label of the significant location. Forexample, the confidence level can be indicated by a marker (for example,by presenting a question mark next to the label or by presenting thelabel in a particular color or other marker). The marker can be editableallowing a user to either confirm that the identified label is accurateor to correct the label. A correction to a label can include re-labelingthe displayed label to be the accurate label. Alternatively or inaddition, the correction can include labeling a different, more accuratesignificant location with the identified label. In response to the userediting the label, the edit is applied to other appearances of the samesignificant location and when the computing device is located at thesignificant location at a future time.

Identifying a Label Based on Payment Transaction

In some implementations, computing device 200 can determine a label fora significant location based on payment transaction data 302 or ahistory of payment transactions stored by a payment applicationimplemented by the computing device 200. The payment transactions caninclude a time of the payment transaction, the location of thetransaction (e.g., point of sale) and an identity (e.g., merchant nameor business identifier) of the significant location at which the paymenttransaction occurred. In some cases, the significant location can be aPOI in the POI database 210. For example, if the user is at a restaurant(labeled “Burger Place”) and executes the payment application on thecomputing device 200 to complete a transaction with the restaurant, thenthe payment application can store a time of the transaction, thelocation of the transaction and the merchant name “Burger Place” aspayment transaction data 302. When the significant location estimator202 provides a significant location, the significant location can beassociated with the MUID for “Burger Place” based on the transactiondata. A marker for the significant location is displayed on the map withthe label “Burger Place.” The label “Burger Place” has meaning to theuser because the user purchased food at the Burger Place one or moretimes in the past.

Identifying a Label Based on Calendar Events

In some implementations, computing device 200 can determine a label fora significant location based on calendar event data 304. The contextualdata sources 214 can include calendar event data 304 received from acalendar application implemented by the computing device 200. Thecalendar event data 304 can include a location field with an address ora text string that includes a “hint” of a location. For example, thetext string could recite “Dinner at Bob's Summer home in the Hamptons.”That text can be used to search a contact or address book for Bob'sSummer home address. A label of a significant location that is nearBob's Summer home can be updated with the exact address found in thecontact or address book. If there is a populated location field, theaddress can be taken from the calendar event data 304.

Identifying a Label Based on Reminders

In some implementations, computing device 200 can determine a label fora significant location based on reminders data 306. A reminder is anapplication that runs on computing device 200 that is configured tonotify a user of computing device 200 about pending projects or tasksthat need to be completed or tracked. The contextual data sources 214can include reminders data 306 received from one or more applicationsimplemented by the computing device 200. For example, the reminders data306 can be received from the calendar application. Alternatively, thereminders data 306 can be received from a notes application executed bythe computing device 200 to create and store a note in reply toinstructions from the user. For example, in response to the userspeaking a voice command, “Remind me to get milk at the grocery store,”the computing device 200 can create a reminder with the text spoken bythe user. In this example, the reminder mentions the significantlocation, namely, “the grocery store”. The context analyzer 212 candetermine that “grocery store” represents a significant location. Forexample, the context analyzer 212 can determine if any POIs near thesignificant location is a grocery store and use the POI label for thegrocery store to label the significant location with a meaningful label.In some cases, there may be multiple grocery store POIs near thesignificant location. In these cases, additional contextual data sources214 can be used to narrow the multiple POIs to a single POI. Forexample, if the user has visited a particular POI more than others inthe past, that contextual data can be used to filter the POIs to asingle POI. In another example, transaction data can be used to filterthe POIs as described above.

Identifying a Label Based on Map Data

In some implementations, computing device 200 can determine a label fora significant location based on map data. The contextual data sources214 can include map data 308 received from a map application implementedby the computing device 200, for example, the maps engine 206.

For example, the map data 308 can include route data 308 a that storesroutes traveled by the computing device 200. The route data 308 a caninclude route information including a destination location. Thedestination location can be a location that the significant locationestimator 202 has previously determined to be a significant location.The context analyzer 212 can determine that the destination location isat or near a POI with an assigned label. Based on this determination,the context analyzer 212 can assign the label of the POI to thesignificant location and store the label in the route data 308 a. Inthis manner, the context analyzer 212 can use the route data 308 a tolabel the significant location.

In another example, the map data 308 can include favorites data 308 bthat stores locations identified as “Favorites” by the user of thecomputing device 200. The locations identified as favorites can besignificant locations. One or more of the significant locations, e.g.,locations identified as favorites, can also be POIs with respectiveassigned labels. The context analyzer 212 can determine that asignificant location identified as a favorite is also a POI. Based onthis determination, the context analyzer 212 can assign the label of thePOI to the significant location and store the label in the favoritesdata 308 b.

In a further example, the map data 308 can include history data 308 cthat stores locations that the user of the computing device 200 viewedusing the maps application or that the user viewed and visited. Forexample, the user can view a significant location in the mapsapplication by selecting a marker on a map representing the significantlocation to view details and perhaps photos of the significant location.A location that the user viewed and visited has a higher likelihood ofbeing a significant location than a location that a user only viewed,especially when the user visited the location multiple times. One ormore of the significant locations, e.g., locations that the user viewed(or viewed and visited), can also be POIs with respective assignedlabels. The context analyzer 212 can determine that a significantlocation that the user viewed (or viewed and visited) is also a POI.Based on this determination, the context analyzer 212 can assign thelabel of the POI to the significant location and store the label in thehistory data 308 c.

In another example, the map data 308 can include third party data 308 d.The computing device 200 can be configured to implement third partyapplications, e.g., applications created by parties other than thosethat developed the computing device 200. Certain third partyapplications can provide locations to the user of the computing device200, for example, in response to a search request. An exemplary thirdparty application is one that provides suggestions for restaurants inresponse to a user searching for restaurants within a geographic area.The maps application can display the geographic location of a restaurantprovided by such a third party application. In addition, the mapsapplication can store location information associated with therestaurant including, for example, the geographic location, therestaurant's name or other location information. The computing device200 can determine that the restaurant provided by the third partyapplication is a significant location, for example, if the user visitsthe restaurant (one or more times) or if the user frequently views therestaurant using the third party application (or both). In this manner,the computing device 200 can identify multiple significant locationsbased on usage of the third party application. One or more of thesignificant locations, e.g., locations that the user viewed (or viewedand visited) using the third party application, can also be POIs withrespective assigned labels. The context analyzer 212 can determine thata significant location that the user viewed (or viewed and visited)using the third party application is also a POI. Based on thisdetermination, the context analyzer 212 can assign the label of the POIto the significant location and store the label in the third party data308 d.

In a further example, accessing a POI on a third party application (forexample, viewing, browsing or selecting the POI) is, by itself, anindication that the POI is relevant. That is, the POI is relevant evenif the map data 308 does not receive the POI from the third partyapplication. In such instances, the POI selected using the third partyapplication can be stored (for example, as “favorite” or “relevant”locations) and be matched against potential POIs at which the computingdevice 200 can be located.

Identifying a Label Based on Communications Data

In some implementations, computing device 200 can determine a label fora significant location based on communications data. The contextual datasources 214 can include communications data 310 received from acommunication application implemented by the computing device 200. Forexample, the communications data can include communications between thecomputing device 200 and at least one other computing device. Thecommunication application can be a telephone application, a text messageapplication, an audio communication application, an email application, adigital assistant application, a video communication application or anyother application in which communications are sent from or received bythe computing device 200.

For example, the communications data 310 can include text data 310 athat stores text messages exchanged between the computing device 200 andat least one other computing device. The text messages can include textthat mentions a significant location. The location mentioned in the textmessage can also be a POI with an assigned label. The context analyzer212 can extract text from the text message and identify a POI (forexample, a business) whose name is the same as or substantially similarto the extracted text. Upon finding a match, e.g., upon determining thata threshold likelihood that the extracted text matches the business nameis satisfied, the context analyzer 212 can assign the label of the POIto the significant location and store the label in the text data 310 a.

In another example, the communications data 310 can include audio data310 b that stores audio messages exchanged between the computing device200 and at least one other computing device. The audio messages caninclude audio exchanged during voice calls, audio included in voicemailmessages (either incoming voicemail message or voicemail messages lefton other devices using the computing device 200), audio included inother communication applications or combinations of them. The audiomessages can include audio that mentions a significant location. Thelocation mentioned in the audio message can also be a POI with anassigned label. The context analyzer 212 can identify a POI (forexample, a business) whose name is the same as or substantially similarto that of a significant location mentioned in the audio message. Uponfinding a match, e.g., upon determining that a threshold likelihood thatthe name in the audio message matches the business name is satisfied,the context analyzer 212 can assign the label of the POI to thesignificant location and store the label in the audio data 310 b.

In a further example, the communications data 310 can include video data310 c that stores video messages (including images, such as video, andaudio) exchanged between the computing device 200 and at least one othercomputing device. The video messages can include images or audio (orboth) that mentions a significant location. The location mentioned inthe video message can also be a POI with an assigned label. The contextanalyzer 212 can identify a POI (for example, a business) whose name isthe same as or substantially similar to that of a significant locationmentioned in the video message. Upon finding a match, e.g., upondetermining that a threshold likelihood that the name in the videomessage matches the business name is satisfied, the context analyzer 212can assign the label of the POI to the significant location and storethe label in the audio data 310 b.

In some implementations, the communication data can include a mention ofa time or a context based, either or both of which can be used toresolve multiple POIs to a single significant location. For example, thecommunication data can include the statement “See you at the Burger Kingaround 8 pm” in any format (e.g., audio, video, text or combinations ofthem). The mention of time—“8 pm”—in the communication can be used tofurther resolve the significant location at which the computing deviceis present. In another example, the communication data can include thestatement “I will eat sushi tonight at 8 pm.” The mention of “sushi” canprovide a context indicating that the computing device will be locatedat a Japanese restaurant at 8 pm. The context, taken alone or incombination with the time, can be used to resolve the significantlocation at which the computing device is present.

Identifying a Label Based on Movement Data

In some implementations, computing device 200 can determine a label fora significant location based on movement data. The contextual datasources 214 can include movement data 312 received from an inertialsensor, GNSS receiver or digital pedometer application or other movementdata source available to the computing device 200. For example, thecomputing device 200 can determine based on GNSS data that the computingdevice 200 is at a location in which POIs are aggregated (such as a mallwith multiple shops, restaurants and a movie theater). The movement ofthe computing device 200 can be tracked between the multiple POIs. Thecontext analyzer 212 can identify a label to be assigned to asignificant location based on the movement (or lack of movement) of thecomputing device 200 between the multiple POIs. For example, a dwelltime of the computing device 200 when the computing device 200 is at themovie theater is likely to be significantly higher than a dwell time ofthe computing device 200 when the computing device 200 is elsewhere(such as at a store or a restaurant). Based on the increased dwell time,the context analyzer 212 can determine that the computing device 200 isat the movie theater. The context analyzer 212 can identify a label forthe movie theater based on the movie theater having also been designatedas a POI, and store the label in movement data 312.

In some implementations, the computing device 200 can include one ormore motion sensors (e.g., accelerometer, gyroscope, magnetometer orother motion sensors). Using the motion sensors or other sensors (orboth), the context analyzer 212 can determine a type of physicalactivity being performed by the user of the computing device 200. Forexample, if the motion sensor indicates oscillation, the speed indicatesrunning and the location of the computing device 200 remains unchanged,then the context analyzer 212 can determine that the user is running ona treadmill. If the context analyzer 212 determines that the potentialsignificant locations at which the computing device 200 can be presentcan include a gym or a travel agency, the context analyzer 212 canincrease a likelihood that the computing device 200 is at the gym, notthe travel agency, based on motion information sensed by the motionsensors or other sensors.

Other Examples

In some implementations, computing device 200 can determine a label fora significant location based on other data 314. The contextual datasources 214 can include other data 312 storing labels assigned tosignificant locations using techniques other than those described here.

For example, the computing device 200 can be located next to two POIs,each having an equal or nearly equal likelihood of being a significantlocation. The context analyzer 212 can determine that one of the twoPOIs is closed at a time at which the computing device 200 is locatednext to the two POIs. Based on this determination, the context analyzer212 can determine that the POI that is open is the significant locationand assign the label of the open POI to the significant location. In afurther example, each context data source can assign a label to asignificant location after multiple visits by the user to the samesignificant location.

Each of the context data sources described above can provide contextualdata points for identifying a label to be associated with a significantlocation. The context analyzer 212 can identify multiple potentialsignificant locations based on the contextual data points. The contextanalyzer 212 can disambiguate and resolve the multiple contextual datapoints to identify one of the multiple potential significant locationsas having the highest likelihood of being the significant location atwhich the computing device 200 is located. For example, the contextanalyzer 212 can resolve the multiple potential significant locations toone significant location if more than half of the contextual data pointsidentify the same POI as being that significant location. In anotherexample, the context analyzer 212 can assign a respective weight to eachpotential significant location. The weights can be the same ordifferent. For example, a contextual data point received from a mapsapplication can be given more weight than a contextual data pointreceived from a text messaging application. To resolve the multiplepotential significant locations to one significant location, the contextanalyzer 212 can execute a weighted algorithm that considers therespective weight given to each contextual data point.

In some implementations, a search engine history, favorites list,reading list or bookmarks may contain contextual data that can be usedto label a significant location. If multiple POIs are near a significantlocation, then the user's search history, favorites list, reading listor bookmarks can be used to filter the POIs. For example, referringagain to the example of FIG. 1, a significant location may be near MOVIETHEATER and BURGER PLACE. Each of these POIs could be the actualsignificant location. However, the user bookmarked BURGER PLACE and/orhas a search history that includes multiple searches on BURGER PLACE. Inthis case, BURGER PLACE could be selected over MOVIE THEATER as beingthe significant location.

In an embodiment, voice commands can be parsed to extract words orphrases that could provide “hints” about which of multiple POIs near asignificant location is the significant location. For example anintelligent personal assistant or knowledge navigator could receive avoice command “navigate to Burger Place” or a voice command “call BurgerPlace”. These voice commands can be contextual data that can be used tofilter POIs.

FIG. 3B is a schematic diagram illustrating an extension of a geographicarea of uncertainty encompassing a significant location. In the examplesdescribed above, data stored in each context data source was describedas identifying a label for one significant location. In the exampleshown, an estimated significant location 350 is surrounded by ageographic area of uncertainty 352. The geographic area of uncertainty352 represents an uncertainty associated with the estimated significantlocation 350. That is, the actual significant location computing device200 can be located anywhere within the geographic area of uncertainty352 represented by the uncertainty. To label the significant location,the computing device 200 can consider all the POIs that reside (eithercompletely or partially) within the geographic area of uncertainty 352.In some implementations, to label the significant location, thecomputing device 200 can consider all the POIs that reside (eithercompletely or partially) within a second geographic area of uncertainty354 that is larger than (for example, by a factor greater than one suchas two) the geographic area of uncertainty 352.

FIG. 4 is a flowchart of an example of a process 400 for identifyinglabels for significant locations. The process 400 can be implemented ascomputer instructions stored on a computer-readable storage medium andexecuted by one or more processors. For example, the process 400 can beimplemented by the device architecture 500 described in reference toFIG. 5. In general, process 400 determines a significant location of thecomputing device. For example, at 402, process 400 obtains one or moresignificant locations. For example, a significant location estimator canprovide a list of significant locations. At 404, for each significantlocation, process 400 identifies one or more POIs near the significantlocation. Each POI represents an entity, such as a business, publiclandmark, school, hospital, etc. Process 400 determines a label for thedetermined significant location based on contextual data associated withthe significant location. At 406, process 400 determines that a POI isthe significant location based on contextual data. Examples ofcontextual data include but are not limited to: transaction data,calendar data, map data, communication data, voice commands, and anyother data that can provide information that can be used to identify aPOI as a significant location. At 408, a label associated with thedetermined POI is used to label the significant location.

Exemplary Determination of Significant Locations

Some examples of techniques to determine significant locations aredescribed in this and the following paragraphs. Details aboutdetermining significant locations are described in U.S. patentapplication Ser. No. 14/502,677 entitled “Location-Based Services ForCalendar Events”, filed on Sep. 30, 2014 and the entire contents ofwhich are incorporated by reference herein in its entirety. Computingdevice 200 can use machine learning and data mining techniques to learnthe past movement of computing device 200. The past movement can berecorded as significant locations visited by computing device 200 andmovement of computing device 200 between the significant locations.Computing device 200 can determine that a place or region is asignificant location upon determining that, with sufficient certainty,computing device 200 has stayed at the place or region for a sufficientamount of time. The amount of time can be sufficient if it satisfiesvarious criteria, for example, when the amount of time satisfies a timelength threshold (for example, X hours) or a frequency threshold (forexample, X minutes per day, Y number of days per week). Records ofmovement of computing device 200 can include a measured or calculatedtime of entry into each significant location and a measured orcalculated time of exit from each significant location. A significantlocation can be associated with multiple entries and exits.

In addition to significant locations, the records of movement caninclude transitions between the significant locations. Each transitionfrom a first significant location to a second significant location canbe associated with a transition begin timestamp indicating a timecomputing device 200 leaves the first significant location and atransition end timestamp indicating a time computing device 200 entersthe second significant location.

Computing device 200 can represent the records of movement as a statemodel. State model can include states, each representing a significantlocation, and transitions, each representing a movement of computingdevice 200 between significant locations. Additional details ofdetermining the state model are described below.

Based on the state model, computing device 200 can determine (1) atransition probability density that, at a given time, computing device200 moves from a given significant location to each other significantlocation, or (2) an entry probability density that computing device 200enters a significant location from a previously unknown or unrepresentedlocation. A pattern analyzer of computing device 200 can determine adaily, weekly, monthly, or annual movement pattern of computing device200 using the state model. A predictive engine of computing device 200can use transition probability density (or entry probability density)and the movement pattern to forecast a significant location thatcomputing device 200 will enter (or stay) at a future time. Computingdevice 200 can then use the forecast to provide predictive userassistance, for example, to assist the user to plan for a future event.

Exemplary Techniques of Constructing a State Model

The computing device 200 can use the learning techniques to determinethe state model. Computing device 200 can sequentially trace locationdata through time (T). Sequentially tracing location data can beperformed by piggybacking on another application to avoid or reduce costof location data collection. For example, computing device 200 cancollect the location data when another service requests location from alocation determination subsystem of computing device 200. Accordingly,collecting the location data can be “free” without having to activatethe location determination subsystem solely for determining a movementpattern of computing device 200.

Computing device 200 can collect multiple locations over time T.Collecting new locations can be ongoing operations. Locations can bepurged based on a variety of policies, including age, user preference orprivacy. The multiple locations can each include latitude, longitude,and altitude coordinates, with a degree of uncertainty in each of these,and be associated with a timestamp indicating a time the correspondinglocation is collected.

Computing device 200 can determine that some of locations belong tolocation clusters that may indicate a significant location. Computingdevice 200 can determine that a location cluster is formed upondetermining that (1) at least a pre-specified threshold number (forexample, two) of consecutive locations are collected; (2) a time span ofthe consecutive locations satisfies a pre-specified threshold timewindow; and (3) these locations are identical, indicating that computingdevice 200 is stationary, or sufficiently close to one another,indicating that computing device 200 is located in a sufficiently smalland defined area during the time the locations are collected.

For example, computing device 200 can determine two location clustersover time T. A first location cluster can include a first subset oflocations collected over a time period [T1, T2] that is longer than athreshold time window (for example, a time window of 45 minutes).Computing device 200 can determine that the first location clusterincludes the locations in the subset upon determining that a variance ofthe locations is low enough to satisfy a variance threshold. Likewise, asecond location cluster can include a second subset of locations, whichare collected within time period [T3, T4]. Computing device 200 candetermine that the second location cluster includes the locations in thesecond subset upon determining that a variance of the locationssatisfies the variance threshold.

An outlier detection mechanism can filter out locations that do notbelong to clusters. For example, computing device 200 can determine thata location is different from locations in the two subsets (for example,based on a distance threshold being exceeded). In addition, computingdevice 200 can determine that no other locations are (1) collectedwithin the threshold time window before or after the location and (2)geographically close to that location. In response, computing device 200can determine that the location is an outlier and discard the location.In addition, if a location in a time period is significantly differentfrom many other locations in the time period, computing device 200 candiscard the different location as an outlier and determine the locationcluster using other locations in the time window. Computing device 200can use the first and second location clusters to determine significantlocations and states of the state model.

In some implementations, one of the conditions for determining alocation cluster is that a time span of the consecutive locationssatisfies a variable threshold time window. The threshold can vary basedon whether computing device 200 has a hint of significance of alocation.

At various times, computing device 200 can be located at differentlocations. The different locations can be far apart from one another,indicating that computing device 200 is moving. Computing device 200 canbe located at the different locations during a continuous period oftime. The different locations can be identical or sufficiently close toone another. Computing device 200 can determine whether the period oftime is sufficiently long such that the different locations form alocation cluster that indicates a significant location, based on whetherthe period of time satisfies a variable threshold. Computing device 200can use various hints to determine the variable threshold.

For example, computing device 200 can search locations where computingdevice 200 visited previously. Computing device 200 can designate as afirst hint a record indicating that computing device 200 previouslyvisited the location at or near the different locations as a first hint.Computing device 200 can examine a user search history performed on orthrough computing device 200. If the user searched for the locationbefore, computing device 200 can designate a search query including anaddress at or near the different locations, or a business located at ornear the different locations, as a second hint. Computing device 200 candesignate a calendar item in a user calendar (for example, anappointment or a meeting) located at or near the different locations asa third hint.

Upon detecting one or more hints, computing device 200 can use a shortertime period, for example, five minutes, as a threshold for determining alocation cluster or significant location. More hints can correspond toshorter threshold. Accordingly, computing device 200 can determine asignificant location upon detecting a location of the computing device,when the short time threshold is satisfied.

If no hint is found, computing device 200 can use a longer time period,for example, 20 minutes, as a threshold for determining a locationcluster or significant location. Accordingly, when no hint is found,computing device 200 can determine a location cluster or significantlocation upon detecting a location of computing device 200, when thelong time threshold is satisfied. In either case, with or without ahint, computing device 200 can determine a significant location in realtime, for example, 5 minutes or 20 minutes after locations converge intoa cluster.

Using the techniques described above, computing device 200 can identifylocation clusters. Computing device 200 can determine significantlocations based on the location clusters.

Computing device 200 can determine each of the significant locationsbased on the location clusters using the locations in each of thelocation clusters. Determining the significant locations can be based onrecursive filter with a constant gain. Details of determining thesignificant locations are provided below. Each of the significantlocations can include latitude, longitude, and optionally, altitudecoordinates. Each of the significant locations can be associated withone or more location clusters. For example, a first significant locationcan correspond to a first location cluster in time period [T1, T2] and athird location cluster during time period [T7, T8]. The location in thefirst location cluster and the third location cluster can be identical.The length of time period [T1, T2] and time window [T7, T8] can be thesame or different.

Computing device 200 can have an initial state model at time T2. At timeT2+k, computing device 200 can receive incremental location data, wherek is a difference between time T2 and the time the additional locationdata are received (in this example, k=T7−T2). Computing device 200 canuse the incremental location data to determine the significant locationfor use in the state model. Computing device 200 can determine that thefirst location cluster corresponds to latitude and longitude coordinatesX1. Computing device 200 can determine that the third location clustercorresponds to latitude and longitude coordinates X2. Computing device200 can determine that a distance between X1 and X2 satisfies athreshold. In response, computing device 200 can determine that thefirst location cluster and the third location cluster belong to a samelocation (e.g., the same significant location). Computing device 200 canthen add the third location cluster to the significant location usingconstant gain filter as shown below in Equation (1).

$\begin{matrix}{\frac{{X\; 2} + {\alpha\; X\; 1}}{1 + \alpha},{{{where}\mspace{14mu}\alpha} \geq 1}} & (1)\end{matrix}$

Each of the significant locations can be associated with one or moreentry timestamps and one or more exit timestamps. Each entry timestampcan correspond to a time associated with a first location in a locationcluster. For example, a first entry timestamp associated with thesignificant location can be a timestamp associated with a location,which is the first location of the first location cluster. A secondentry timestamp associated with a significant location can be atimestamp associated with a first location in the third locationcluster. Likewise, each exit timestamp can correspond to a timeassociated with a last location in a location cluster. For example, afirst exit timestamp associated with the first significant location canbe a timestamp associated with the location, which is the last locationof the first location cluster. A second entry timestamp associated withthe significant location can be a timestamp associated with a lastlocation in the third location cluster.

Example Location Label Harvesting System

As previously described, computing device 200 is capable of labelinglocations including locations that are significant to a user ofcomputing device 200. Once computing device 200 obtains a label for alocation, the location label and a wireless access point data isharvested or “crowdsourced” by one or more server computers and used toupdate a database (or other data structure) coupled to the servercomputers. In an embodiment, multiple computing devices 200 monitortheir respective significant locations and observe wireless signalstransmitted from wireless access points (e.g., WiFi access points) atthe respective significant locations. The monitoring can be a backgroundprocess that is transparent to the user. In some embodiments, themonitoring can be performed by a low-power wireless baseband processorof the computing device rather than a central processing unit of thecomputing device that consumes more power than the baseband processor.The location label and wireless access point data are cached locally onthe computing device 200, as described in reference to FIG. 5. In anembodiment, the cached data is persisted on the computing device 200until a time when the cached data can be uploaded to the one or moreserver computers (an “upload opportunity”). An example uploadopportunity is when the computing device is online or in communicationwith a network server computer and the network has sufficient bandwidthto handle the upload and the computing device has sufficient power toperform the upload. For example, sufficient bandwidth and power meanscomputing device 200 has enough processing cycles and power (e.g.,battery power) to establish and maintain a communication session withthe one or more server computers through a wired or wirelesscommunication network.

A requesting computing device operating at significant location can usewireless access point data (e.g., WiFi scan data) to obtain a locationlabel from the service to be used by, for example, a mapping ornavigation application running on the requesting device. For example, ifa harvested computing device learns a discrete significant location andis able to label the significant location as “Big Al's Pizza,” then therequesting device can benefit from the location label that was providedby the harvested computing device.

FIG. 5 is a conceptual diagram of an example harvesting system 500 forharvesting location labels. System 500 can include network 502, one ormore server computers 504 and client devices 506 a,506 b. Client devices506 a, 506 b are computing devices from which location data is harvestedas previously described in reference to FIGS. 1-4 (hereafter alsoreferred to as “harvested devices”). Client devices 506 a, 506 b cancouple to network 502 through a variety of wireless and wired accesspoints 507 a, 507 b including but not limited to wireless access points(e.g., WiFi access points), cell towers and wired network connections(e.g., Ethernet). Network 502 can include one or more networksincluding, for example, the Internet. Client devices 506 a, 506 b canoperate at different geographic locations throughout the world. One ormore databases 508 can be coupled to the one or more server computers504 for storing location labels and other location data.

An example of location label harvesting will now be described inreference to FIG. 5. Client devices 506 a are operating at location Aand client devices 506 b are operating at location B. For each of clientdevices 506 a, location A is a learned significant location. Clientdevices 506 a can communicate with server computer(s) 504 through accesspoint 507 a and network 502. Similarly, for each of client devices 506b, location B is a learned significant location. Client devices 506 bcan communicate with server computer(s) 504 through access point 507 band network 502. The significant location can be learned and labeledusing the methods described in reference to FIGS. 1-4. Although twolocations and four client devices are described, in practice any numberof locations and client devices can be used.

In an embodiment, the harvesting can be triggered by a transactionevent, where client devices 506 a, 506 b are used to pay for a productor service at a point of sale. Transaction event data (e.g., atimestamp, description, geographic location of the point of sale) isstored on client devices 506 a, 506 b. Wireless access point data (e.g.,a WiFi scan data) and motion classification data are also obtained andstored on client devices 506 a, 506 b. Motion classification data caninclude information describing the motion of the client device based onsensor data provided by one or more sensors (e.g., accelerometer, gyro,magnetometer) onboard or coupled to the client device. Some examples ofmotion classification data can include but are not limited to: walking,running, driving and biking. In an embodiment, raw sensor data can bestored in addition to, or in lieu of, the motion classification data.Motion classification data can be useful for determining whether a useris walking into a significant location (e.g., a retail store) ratherthan driving by a significant location in a car.

In an embodiment, the wireless access point data can include receivedsignal strength indicator (RSSI) values (or some other signal strengthmetrics) for RF signals received from the wireless access points at thelocation. In an embodiment, the RF signals can be WiFi signals andclient devices 506 a, 506 b can generate RSSI values from results ofWiFi scans. The RSSI values can be associated with timestamps andhardware identifiers for the WiFi access points such as media accesscontrol (MAC) addresses. Hereafter, the collection of RSSI valuesobserved by a client device at a particular location are also referredto as a “fingerprint” of that location.

In an embodiment, the transaction event data, wireless access point dataand motion classification data (and/or raw sensor data) are storedcontinuously in one or more ring buffers (collectively referred to as“location data”). In an embodiment, when a transaction event occurs, thelocation data is collected from the ring buffer(s) for a time periodbefore, during and after the transaction event timestamp. For example,location data can be collected from the ring buffer(s) 30 seconds beforethe transaction event begins and 30 seconds after the transaction eventends based on, for example, a timestamp of the transaction event.

In this harvesting example, each of client devices 506 a, 506 b haspreviously obtained a location label using contextual data, as describedin reference to FIGS. 1-4 and the label is stored as part of thelocation data by each client device 506 a, 506 b. Client devices 506 a,506 b upload the location data to server computer(s) 504 through accesspoints 507 a, 507 b and network 502, where the location data is storedby server computer(s) 504 in database 508. In an embodiment, clientdevices 506 a, 506 b look for an upload opportunity (e.g., connection toserver computer(s) 504) and perform the upload using a backgroundprocess transparent to the user of the client device. In an embodiment,the upload is incremental to conserve power on the client device.

At the server computer(s) 504, the RSSI values included in the locationdata are used to update a probability distribution of RSSI values forthe given location. In an embodiment, the location can be a cell in avirtual grid overlying a geographic space (also referred to as a radiomap) and the probability distribution of RSSI values can be associatedwith the cell. When the probability distribution for the cell is updatedwith the RSSI values, the location label that was included in thelocation data is associated with the cell in database 508.

When a requesting device is operating at location A or location B, arequest or query is sent to server computer(s) 504 that includeswireless access point data. The wireless access point data (e.g., RSSIvalues) are compared to fingerprints stored in database 508. Eachfingerprint can be associated with geographic coordinates (e.g.,latitude, longitude, altitude) in a deterministic or statistical manner,as described in further detail in reference to FIG. 9. When a match witha fingerprint is determined, server computer(s) 504 respond to therequest/query with the location label associated with the matchedfingerprint. The request/query can be made by, for example, anapplication running on the requesting device such as a mapping ornavigation application. The location label can then be used to labellocation A or location B on a map provided by the mapping application ora navigation application.

By harvesting labels for locations with fingerprints, discrete indoorsignificant locations can be identified with labels meaningful to theuser. For example, “Big Al's Pizza” (an indoor discrete location) couldbe a restaurant located within “ACME Shopping Mall.” By associatinglabels with fingerprints stored in the one or more databases 508, thediscrete location label “Big Al's Pizza” can be shown on a map ratherthan the non-discrete location label “ACME Shopping Mall.” Therefore,the user's location is represented by a marker on the map with a labelthat is more precise and therefore more informative to the user.

In some embodiments, in addition to wireless access point data otherenvironment data can be sent to server computer(s) 504 to assist inidentifying a label for a location. Such environment data can includebut is not limited to: data representing ambient light or ambient noise,client device sensor data (e.g., accelerometer data, gyro data,magnetometer data), network identifiers, temperature, barometricpressure and any other data that can be used to associate a locationlabel with a fingerprint.

Example Label Harvesting Processes

FIG. 6 is a flowchart of an example process 600 for harvesting labelsfor locations performed by a client device. Process 600 can be performedusing device architecture 1000, as described in reference to FIG. 10.

Process 600 can begin by obtaining, by a client device, a label for alocation of the client device (602). For example, contextual data can beused to determine a meaningful label of a signification location, asdescribed in reference to FIGS. 1-4.

Process 600 can continue by obtaining, by the client device, wirelessaccess point data (604). For example, before, during and after atransaction event, a wireless transceiver operating on the client devicecan calculate RSSI values for radio frequency signals transmitted by aplurality of wireless access points at the location (e.g., location A orB). The RSSI values and other wireless access point data (timestamp, MACaddresses) are stored on the client device in, for example, one or morering buffers. Also, transaction data can also be stored on the clientdevice. The transaction data can include data associated with thelocation A or B (a point of sale of a good or service) and include atimestamp, description (e.g., store description or label) and possibly ageographic location or a “hint” of a geographic location. For example, a“hint” can include the name of a city, street or neighborhood in thename of the product or service provider, such as “North Beach Pizza,”which may be found in transaction event data, such as an electronicreceipt.

Process 600 can continue by determining an upload opportunity (606) forthe cached location data. For example, the client device can determinethat a connection has been established with server computer(s) 504 andthat there is sufficient network bandwidth and client device processingpower to perform an upload. The upload can be incremental in that onlydata that has changed since a last upload is updated.

Process 600 can continue by uploading (608) the harvested location datato server computer(s) 504. For example, the location data can be sent inone or more packets to server computer(s) 504.

FIG. 7 is a flowchart of an example process 700 for harvesting labelsfor locations performed by one or more server computers. Process 700 canbegin by receiving, by server computer(s), harvested location dataincluding wireless access point data and a location label from aplurality of client devices (702). In an embodiment, the wireless accesspoint data can include but is not limited to: estimated device location,RSSI values observed, MAC addresses, timestamps and motion data (e.g.,acceleration data, compass data). The estimated device location can bedetermined by the client device using GNSS data (if outdoors andavailable) or it can be provided by a user using a surveying devicebased on a floorplan or other survey data.

Process 700 continues by associating the harvested location data with afingerprint (704) and updating the fingerprint with the harvestedlocation data in a database or other data structure (706). For example,a probability distribution of RSSI values associated with the locationcan be updated by the harvested RSSI values. Additionally, if no labelis associated with the WiFi fingerprint, then the WiFi fingerprint canbe associated with the harvested location label in the database or otherdata structure. In an embodiment, if a conflict of location labels isobserved (e.g., different label descriptions) over multiple updates,server computer(s) 504 can create a histogram or other statisticalconstruct of received location labels from client devices and select thelocation label with the highest frequency or count as the best locationlabel.

In an embodiment, other environment fingerprints can be updated orassociated with the location. For example, “Big Al's Pizza” could belocated on a first floor of the mall and the user is located at a storeon a second floor of the mall above Big Al's Pizza. The wireless accesspoint data may not be discrete enough to determine the store at whichthe user is located. In such a scenario, barometric pressure data oraltitude data could be used to determine the correct discrete locationof the user. Such information can be combined with WiFi scan data tocreate an environment fingerprint.

FIG. 8 is a flowchart of an example process 800 for serving labels forlocations performed by one or more server computers. Process 800 canbegin by receiving a request/query for location and/or POI data from arequesting device that includes wireless access point data (802). Therequest/query can be made by an operating system or application (e.g., amapping application or search engine) running on the requesting device.The wireless access point data can include, for example, a fingerprint(e.g., RSSI values and MAC addresses) collected by a wireless scan atthe location. The request/query can also include a location estimatecalculated by the requesting device using, for example, the wirelessaccess point data and a radio map provided by the one or more servercomputers. Process 800 then matches the fingerprint received with therequest/query with a labeled fingerprint in a database coupled to theone or more servers (804), as described in reference to FIG. 9. Process800 can continue by obtaining a location label associated with afingerprint in the database (806) and then serving the obtained locationlabel to the requesting device (808).

An example database is shown in Table I.

TABLE I Example WiFi Database with Location Labels RSSI PDF LocationValues Parameters MAC addresses Timestamps Labels Location RSSI[0] Mean,MAC_addr[0] Timestamp[0] Label Lat, Lon, Alt RSSI[1] VarianceMAC_addr[1] Timestamp[0] RSSI[2] MAC_addr[2] Timestamp[0] RSSI[0] Mean,MAC_addr[0] Timestamp[0] Label Lat, Lon, Alt RSSI[1] VarianceMAC_addr[1] Timestamp[1] RSSI[2] MAC_addr[2] Timestamp[2] RSSI[3]MAC_addr[3] Timestamp[3] RSSI[0] Mean, MAC_addr[0] Timestamp[0] LabelLat, Lon, Alt RSSI[1] Variance MAC_addr[1] Timestamp[1]

The example WiFi database shows 3 example records. Each record includesa fingerprint, which in this case includes RSSI values for observedaccess points, MAC addresses of the observed access points, timestampsof when the observations were made and a location label associated withthe fingerprint. Note that for each example record there can be anynumber of access point observations depending on the number of accesspoints at the location. In the example shown, the location labels areincluded with the location information but could also be in a separatedatabase. The fingerprint matching can be either deterministic orprobabilistic. For probabilistic fingerprint matching, statistics (e.g.mean, variance) can be included in the database for determining theprobability density function of RSSI values at the location can bestored, as described in reference to FIG. 9.

FIG. 9 is a conceptual block diagram of an example system 900implemented by one or more server computers for harvesting and servinglabels for locations. In the example embodiment shown, system 900includes server 902, filtering module 904, fingerprinting module 906 anddatabase 908.

Server 902 provides an interface to client devices and requestingdevices. Server 909 is coupled to network 102 and transmits and receivespackets to and from client devices and requesting devices. Filteringmodule 904 culls out bad location data that is determined to be outlierdata so that the location data does not infect the fingerprints. Forexample, fingerprints associated with moving access points areidentified and discarded based on, for example, motion classificationdata and/or raw sensor data. Also, outlier data that violatesconsistency checks is discarded. For example, due to severe multipath atsome harvesting locations one or more RSSI values may be substantiallydifferent than other RSSI values possibly indicating that the RSSI valueis corrupted.

In an embodiment, fingerprinting module 906 performs at least twofunctions. The first function is to use the harvested location data toupdate the fingerprint database 908 by, for example, updatingfingerprints with location labels. The second function is to processrequests/queries from requesting devices (e.g., from a mappingapplication or operating system service running on the requestingdevice) to obtain location labels for the locations and to respond torequest/query with the estimated location and location label.

For the second function, database 908 includes radio maps of areas basedon previously harvested wireless access point data, such as RSSI valuesdata calculated from wireless access point transmissions observed atmultiple reference locations in the areas and generates a probabilitydistribution of RSSI values for each reference location. RSSI valuesobtained by a requesting device are then compared to the fingerprint tofind the closest match and generate an estimated location of therequesting device. Fingerprinting module 906 takes the live RSSI valuesas input and generates data necessary to build or update the RSSIprobability distribution for each reference location in the area. In anembodiment, the area can be conceptually divided into a grid of cells,where a probability distribution of RSSI values is generated for eachcell in the grid.

In some embodiments, a Markov localization or Bayes' theorem can be usedto estimate or predict the most likely location of the requestingdevice. Using Markov localization, a prediction step can be representedmathematically by Equation [1]:p(L _(t))=Σ_(Lt-1) p(L _(t) |L _(t-1))p(L _(t-1)),  [1]where p(L_(t)) is the probability of being at location L at time t andp(L_(t)|L_(t-1)) is the probability of being at location L at time tgiven the previous location L at time t−1. The prediction step is thenfollowed by a correction step given by Equation [2]:p(L _(t) |{right arrow over (R)})=p({right arrow over (R)}|L _(t))p(L_(t))*N,  [2]where p(L_(t)|{right arrow over (R)}) is the probability of being atlocation L at time t given the vector of RSSI values R received at timet, p({right arrow over (R)}|L_(t))p(L_(t)) is the probabilitydistribution generated from the harvested RSSI values, p(L_(t)) is theprobability of being at that location from Equation [1] and N is anormalization factor.

Equation [2] can be calculated for every fingerprint in the radio mapfor the area and the fingerprint that yields the maximum probabilityp(L_(t)|{right arrow over (R)}) is the matched fingerprint. The locationlabel associated with the matched fingerprint in database 908 isdownloaded to the requesting device from server computer(s) 504.

In another embodiment, the most probable location l given an observationvector of {right arrow over (S)} of k RSSI values can be determinedusing Bayes' theorem as shown in Equation [3]:argmax_(l) [P(l|{right arrow over (S)})]=argmax_(l)[Π_(i=1) ^(k) P(s_(i) |l)],  [3]where P(s_(i)|l) is the probability distribution of RSSI values forreference location l in the area. Equation [3] assumes that allreference locations l in the area are equally likely and the probabilityof observing {right arrow over (S)} or P({right arrow over (S)}) isconstant for all reference locations l. The location label in database908 associated with the most probable location l as determined byEquation [3] is downloaded to the requesting device from servercomputer(s) 504.

Exemplary Device Architecture

FIG. 10 is a block diagram of example device architecture forimplementing the features and processes described in reference to FIGS.1-9. Architecture 1000 may be implemented in any computing device forgenerating the features and processes described in reference to FIGS.1-9, including but not limited to smart phones and wearable computers(e.g., smart watches, fitness bands). Architecture 1000 may includememory interface 1002, data processor(s), image processor(s) or centralprocessing unit(s) 1004, and peripherals interface 1006. Memoryinterface 1002, processor(s) 1004 or peripherals interface 1006 may beseparate components or may be integrated in one or more integratedcircuits. One or more communication buses or signal lines may couple thevarious components.

Sensors, devices, and subsystems may be coupled to peripherals interface1006 to facilitate multiple functionalities. For example, motion sensor1010, light sensor 1012, and proximity sensor 1014 may be coupled toperipherals interface 1006 to facilitate orientation, lighting, andproximity functions of the device. For example, in some implementations,light sensor 1012 may be utilized to facilitate adjusting the brightnessof touch surface 1046. In some implementations, motion sensor 1010(e.g., an accelerometer, rate gyroscope) may be utilized to detectmovement and orientation of the device. Accordingly, display objects ormedia may be presented according to a detected orientation (e.g.,portrait or landscape).

Other sensors may also be connected to peripherals interface 1006, suchas a temperature sensor, a barometer, a biometric sensor, or othersensing device, to facilitate related functionalities. For example, abiometric sensor can detect fingerprints and monitor heart rate andother fitness parameters.

Location processor 1015 (e.g., GNSS receiver chip) may be connected toperipherals interface 1006 to provide geo-referencing. Electronicmagnetometer 1016 (e.g., an integrated circuit chip) may also beconnected to peripherals interface 1006 to provide data that may be usedto determine the direction of magnetic North. Thus, electronicmagnetometer 1016 may be used as an electronic compass.

Camera subsystem 1020 and an optical sensor 1022, e.g., a chargedcoupled device (CCD) or a complementary metal-oxide semiconductor (CMOS)optical sensor, may be utilized to facilitate camera functions, such asrecording photographs and video clips.

Communication functions may be facilitated through one or morecommunication subsystems 1024. Communication subsystem(s) 1024 mayinclude one or more wireless communication subsystems. Wirelesscommunication subsystems 1024 may include radio frequency receivers andtransmitters and/or optical (e.g., infrared) receivers and transmitters.Wired communication systems may include a port device, e.g., a UniversalSerial Bus (USB) port or some other wired port connection that may beused to establish a wired connection to other computing devices, such asother communication devices, network access devices, a personalcomputer, a printer, a display screen, or other processing devicescapable of receiving or transmitting data.

The specific design and implementation of the communication subsystem1024 may depend on the communication network(s) or medium(s) over whichthe device is intended to operate. For example, a device may includewireless communication subsystems designed to operate over a globalsystem for mobile communications (GSM) network, a GPRS network, anenhanced data GSM environment (EDGE) network, IEEE802.xx communicationnetworks (e.g., Wi-Fi, Wi-Max, ZigBee™), 3G, 4G, 4G LTE, code divisionmultiple access (CDMA) networks, near field communication (NFC), Wi-FiDirect and a Bluetooth™ network. Wireless communication subsystems 1024may include hosting protocols such that the device may be configured asa base station for other wireless devices. As another example, thecommunication subsystems may allow the device to synchronize with a hostdevice using one or more protocols or communication technologies, suchas, for example, TCP/IP protocol, HTTP protocol, UDP protocol, ICMPprotocol, POP protocol, FTP protocol, IMAP protocol, DCOM protocol, DDEprotocol, SOAP protocol, HTTP Live Streaming, MPEG Dash and any otherknown communication protocol or technology.

Audio subsystem 1026 may be coupled to a speaker 1028 and one or moremicrophones 1030 to facilitate voice-enabled functions, such as voicerecognition, voice replication, digital recording, and telephonyfunctions.

I/O subsystem 1040 may include touch controller 1042 and/or other inputcontroller(s) 1044. Touch controller 1042 may be coupled to touchsurface 1046. Touch surface 1046 and touch controller 1042 may, forexample, detect contact and movement or break thereof using any of anumber of touch sensitivity technologies, including but not limited to,capacitive, resistive, infrared, and surface acoustic wave technologies,as well as other proximity sensor arrays or other elements fordetermining one or more points of contact with touch surface 1046. Inone implementation, touch surface 1046 may display virtual or softbuttons and a virtual keyboard, which may be used as an input/outputdevice by the user.

Other input controller(s) 1044 may be coupled to other input/controldevices 1048, such as one or more buttons, rocker switches, thumb-wheel,infrared port, USB port, and/or a pointer device such as a stylus. Theone or more buttons (not shown) may include an up/down button for volumecontrol of speaker 1028 and/or microphone 1030.

In some implementations, device 1000 may present recorded audio and/orvideo files, such as MP3, AAC, and MPEG video files. In someimplementations, device 1000 may include the functionality of an MP3player and may include a pin connector for tethering to other devices.Other input/output and control devices may be used.

Memory interface 1002 may be coupled to memory 1050. Memory 1050 mayinclude high-speed random access memory or non-volatile memory, such asone or more magnetic disk storage devices, one or more optical storagedevices, or flash memory (e.g., NAND, NOR). Memory 1050 may storeoperating system 1052, such as Darwin, RTXC, LINUX, UNIX, OS X, iOS,WINDOWS, or an embedded operating system such as VxWorks. Operatingsystem 1052 may include instructions for handling basic system servicesand for performing hardware dependent tasks. In some implementations,operating system 1052 may include a kernel (e.g., UNIX kernel).

Memory 1050 may also store communication instructions 1054 to facilitatecommunicating with one or more additional devices, one or more computersor servers, including peer-to-peer communications. Communicationinstructions 1054 may also be used to select an operational mode orcommunication medium for use by the device, based on a geographiclocation (obtained by the GPS/Navigation instructions 1068) of thedevice.

Memory 1050 may include graphical user interface instructions 1056 tofacilitate graphic user interface processing, including a touch modelfor interpreting touch inputs and gestures; sensor processinginstructions 1058 to facilitate sensor-related processing and functions;phone instructions 1060 to facilitate phone-related processes andfunctions; electronic messaging instructions 1062 to facilitateelectronic-messaging related processes and functions; web browsinginstructions 1064 to facilitate web browsing-related processes andfunctions; media processing instructions 1066 to facilitate mediaprocessing-related processes and functions; GPS/Navigation instructions1068 to facilitate GPS and navigation-related processes and functions;camera instructions 1070 to facilitate camera-related processes andfunctions; and other instructions 1072 for implementing the features andprocesses, as described in reference to FIGS. 1-9.

Each of the above identified instructions and applications maycorrespond to a set of instructions for performing one or more functionsdescribed above. These instructions need not be implemented as separatesoftware programs, procedures, or modules. Memory 1050 may includeadditional instructions or fewer instructions. Furthermore, variousfunctions of the device may be implemented in hardware and/or insoftware, including in one or more signal processing and/or applicationspecific integrated circuits (ASICs).

The features described may be implemented in digital electroniccircuitry or in computer hardware, firmware, software, or incombinations of them. The features may be implemented in a computerprogram product tangibly embodied in an information carrier, e.g., in amachine-readable storage device, for execution by a programmableprocessor; and method steps may be performed by a programmable processorexecuting a program of instructions to perform functions of thedescribed implementations by operating on input data and generatingoutput.

The described features may be implemented advantageously in one or morecomputer programs that are executable on a programmable system includingat least one programmable processor coupled to receive data andinstructions from, and to transmit data and instructions to, a datastorage system, at least one input device, and at least one outputdevice. A computer program is a set of instructions that may be used,directly or indirectly, in a computer to perform a certain activity orbring about a certain result. A computer program may be written in anyform of programming language (e.g., Objective-C, Java), includingcompiled or interpreted languages, and it may be deployed in any form,including as a stand-alone program or as a module, component,subroutine, or other unit suitable for use in a computing environment.

Suitable processors for the execution of a program of instructionsinclude, by way of example, both general and special purposemicroprocessors, and the sole processor or one of multiple processors orcores, of any kind of computer. Generally, a processor will receiveinstructions and data from a read-only memory or a random access memoryor both. The essential elements of a computer are a processor forexecuting instructions and one or more memories for storing instructionsand data. Generally, a computer may communicate with mass storagedevices for storing data files. These mass storage devices may includemagnetic disks, such as internal hard disks and removable disks;magneto-optical disks; and optical disks. Storage devices suitable fortangibly embodying computer program instructions and data include allforms of non-volatile memory, including by way of example, semiconductormemory devices, such as EPROM, EEPROM, and flash memory devices;magnetic disks such as internal hard disks and removable disks;magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor andthe memory may be supplemented by, or incorporated in, ASICs(application-specific integrated circuits). To provide for interactionwith a user the features may be implemented on a computer having adisplay device such as a CRT (cathode ray tube), LED (light emittingdiode) or LCD (liquid crystal display) display or monitor for displayinginformation to the author, a keyboard and a pointing device, such as amouse or a trackball by which the author may provide input to thecomputer.

One or more features or steps of the disclosed embodiments may beimplemented using an Application Programming Interface (API). An API maydefine one or more parameters that are passed between a callingapplication and other software code (e.g., an operating system, libraryroutine, function) that provides a service, that provides data, or thatperforms an operation or a computation. The API may be implemented asone or more calls in program code that send or receive one or moreparameters through a parameter list or other structure based on a callconvention defined in an API specification document. A parameter may bea constant, a key, a data structure, an object, an object class, avariable, a data type, a pointer, an array, a list, or another call. APIcalls and parameters may be implemented in any programming language. Theprogramming language may define the vocabulary and calling conventionthat a programmer will employ to access functions supporting the API. Insome implementations, an API call may report to an application thecapabilities of a device running the application, such as inputcapability, output capability, processing capability, power capability,communications capability, etc.

As described above, some aspects of the subject matter of thisspecification include gathering and use of data available from varioussources to improve services a computing device can provide to a user.The present disclosure contemplates that in some instances, thisgathered data may identify a particular location or an address based ondevice usage. Such personal information data can include location baseddata, addresses, subscriber account identifiers, or other identifyinginformation.

The present disclosure further contemplates that the entitiesresponsible for the collection, analysis, disclosure, transfer, storage,or other use of such personal information data will comply withwell-established privacy policies and/or privacy practices. Inparticular, such entities should implement and consistently use privacypolicies and practices that are generally recognized as meeting orexceeding industry or governmental requirements for maintaining personalinformation data private and secure. For example, personal informationfrom users should be collected for legitimate and reasonable uses of theentity and not shared or sold outside of those legitimate uses. Further,such collection should occur only after receiving the informed consentof the users. Additionally, such entities would take any needed stepsfor safeguarding and securing access to such personal information dataand ensuring that others with access to the personal information dataadhere to their privacy policies and procedures. Further, such entitiescan subject themselves to evaluation by third parties to certify theiradherence to widely accepted privacy policies and practices.

In the case of advertisement delivery services, the present disclosurealso contemplates embodiments in which users selectively block the useof, or access to, personal information data. That is, the presentdisclosure contemplates that hardware and/or software elements can beprovided to prevent or block access to such personal information data.For example, in the case of advertisement delivery services, the presenttechnology can be configured to allow users to select to “opt in” or“opt out” of participation in the collection of personal informationdata during registration for services.

Therefore, although the present disclosure broadly covers use ofpersonal information data to implement one or more various disclosedembodiments, the present disclosure also contemplates that the variousembodiments can also be implemented without the need for accessing suchpersonal information data. That is, the various embodiments of thepresent technology are not rendered inoperable due to the lack of all ora portion of such personal information data. For example, content can beselected and delivered to users by inferring preferences based onnon-personal information data or a bare minimum amount of personalinformation, such as the content being requested by the deviceassociated with a user, other non-personal information available to thecontent delivery services, or publicly available information.

A number of implementations have been described. Nevertheless, it willbe understood that various modifications may be made. Elements of one ormore implementations may be combined, deleted, modified, or supplementedto form further implementations. In yet another example, the logic flowsdepicted in the figures do not require the particular order shown, orsequential order, to achieve desirable results. In addition, other stepsmay be provided, or steps may be eliminated, from the described flows,and other components may be added to, or removed from, the describedsystems. Accordingly, other implementations are within the scope of thefollowing claims.

The invention claimed is:
 1. A method comprising: receiving, by one ormore server computers, location data including wireless access pointdata and location labels, the location data being harvested from aplurality of devices operating at a plurality of geographic locations,each location label indicating an association with a respectivesignificant location as determined by a respective device and comprisinga description of the significant location as determined by therespective device; and updating, by the one or more server computers, aplurality of fingerprints representing the plurality of geographiclocations, the updating including associating at least one of thereceived location labels with at least one of the plurality offingerprints.
 2. The method of claim 1, wherein the location datafurther includes motion classification data classifying a movementdetected by a respective device as a particular user activity type fromamong a plurality of user activity types, and the updating furthercomprises associating at least one of the received location labels withat least one the plurality of fingerprints using the motionclassification data.
 3. The method of claim 1, wherein the location datafurther includes commercial transaction data associated with aparticular significant location, and the updating further comprisesassociating the at least one received location label with the at leastone fingerprint using the commercial transaction data.
 4. The method ofclaim 1, wherein the wireless access point data includes received signalstrength values calculated from wireless radio frequency signalstransmitted by one or more wireless access points at the significantlocations.
 5. A system comprising: one or more processors; memorycoupled to the one or more processors and operable for storinginstructions, which, when executed by the one or more processors, causesthe one or more processors to perform operations comprising: receivinglocation data including wireless access point data and location labels,the location data being harvested from a plurality of devices operatingat a plurality of geographic locations, each location label indicatingan association with a respective significant location as determined by arespective device and comprising a description of the significantlocation as determined by the respective device; and updating aplurality or fingerprints representing the plurality of geographiclocations, the updating including associating at least one of thereceived location labels with at least one of the plurality offingerprints.
 6. The system of claim 5, wherein the location datafurther includes motion classification data classifying a movementdetected by a respective device as a particular user activity type fromamong a plurality of user activity types, and the updating furthercomprises associating at least one of the received location labels withat least one the plurality of fingerprints using the motionclassification data.
 7. The system of claim 5, wherein the location datafurther includes commercial transaction data associated with aparticular significant location, and the updating further comprisesassociating the at least one received location label with the at leastone fingerprint using the commercial transaction data.
 8. The system ofclaim 5, wherein the wireless access point data includes received signalstrength values calculated from wireless radio frequency signalstransmitted by one or more wireless access points at the significantlocations.
 9. The method of claim 2, wherein the plurality of useractivity types comprises walking, running, driving, or hiking.
 10. Themethod of claim 3, wherein the commercial transaction data indicatesthat a commercial transaction occur at the particular significantlocation.